Complexity Science (CSS)

• The authors identify that the main focus of CSS is understanding the emergence of macro-level behavior from simple micro-level processes. Mostly qualitative, conceptual models.

• Usually in low-dimensional environments

• High emphasis in mechanism simplicity

• Computationally lightweight

• Cooperation is usually “baked in” the model. An explicit model component is added to capture it.

• Usual toolset

  • Dynamics
  • Analytics

Multi-Agent Reinforcement Learning (MARL)

• Main focus: Improve the cooperation mechanics (then study them)

• Usually very high-dimensional state spaces

• Cooperation is typically not given as an “RL action”, it emerges as a learned behavior

• Very computationally expensive

• Usual toolset:

  • Simulations
  • Algorithm design

• Inherent stochasticity & large number of parameters makes interpretation hard

• We want to use the CSS toolset to understand MARL as a complex dynamical system,
  • and then study the emergence of cooperation under that lens

CRLD strengths

The CSS techniques and approaches are what allow CRLD to bring more insight to MARL. What the CSS bring to the table

• Complex phenomena: CRLD has the power to uncover the emergent behavior at a much lighter computational load

• Multistability: CRLD can characterize entire dynamics, like

  • Phase spaces, with stability landscape (where the equilibria are)
  • Existance of stable/unstable equilibrium points
  • Basins of attraction

• Critical transitions: robust way to find hysteretic behavior.

• Collective memory: Hysterisis observed in MARL is a form of collective memory

  • similarly, we can model potential outsourcing of cognitive functions (e.g. standards, cultural norms & institutions).

Now we focus on what the MARL techniques can offer to enrich the CSS approach

• Cooperation from individual cognition: CSS models the spread of successful strategies as copying. This assumes all agents share success criteria. MARL allows for more realism (e.g. intrinsic motivations, homestasis)

• Cooperation in large collectives: CSS must simplify individuals into a homogenous category. The CRLD approach allows a collective of heterogenous individuals to be treated as a dynamical system. It extends the “mean-field approaches” in MARL

• Cooperation in dynamic environments: MARL naturally works with dynamic, uncertain, partially-observable environments, which is significantly more challenging in CSS.

• The authors believe the study of collective cooperation (intelligent actors in complex environments acting to improve their joint well-being) is critical for a future with collective existential threats (foreshadowing).

The ecological tipping environment

• The ecological tipping environment exhibits hysteresis

• Multistability in CRLD applied to the ecological tipping environment. The dynamics equations used are based on temporal-difference learning \[ X_{t+1}^i (s,a) = \frac{1}{\xi_{X_t}^i (s)} X_{t}^i(s, a) \exp(\eta^i \cdot \delta_{x_t}^i(s, a)) \]

Misc. observations

• Hysteresis is exhibited in the ecological tipping environment when the discount factor is varied
  • Modifications to the discount factor can lead to a critical transition from cooperation to defection, and back, along different paths
• The ecological tipping environment also exhibits timescale separation
  • Agents’ convergence speed to a strategy shows a critical point w.r.t the discount factor

Conclusion

• CRLD tries to leverage the realism of MARL, and the analytical tooling from CSS to understand emergent behaviour in multi-agent systems

• CRLD can capture phenomena such as multistability, critical transitions, and collective memory

• The paper reads as half review, half position paper. The call to action is to try to apply CRLD to as many things as possible